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Abstract 

Background: Health care providers play a significant role in large scale health emergency planning, detection, 
response, recovery and communication with the public. The effectiveness of health care providers in emergency 
preparedness and response roles depends, in part, on public health agencies communicating information in a way 
that maximizes the likelihood that the message is delivered, received, deemed credible and, when appropriate, 
acted on. However, during an emergency, health care providers can become inundated with alerts and advisories 
through numerous national, state, local and professional communication channels. We conducted an alert fatigue 
study as a sub-study of a larger randomized controlled trial which aimed to identify the most effective methods of 
communicating public health messages between public health agencies and providers. We report an analysis of the 
effects of public health message volume/frequency on recall of specific message content and effect of rate of 
message communications on health care provider alert fatigue. 

Methods: Health care providers enrolled in the larger study (n=528) were randomized to receive public health 
messages via email, fax, short message service (SMS or cell phone text messaging) or to a control group that did 
not receive messages. For 12 months, study messages based on real events of public health significance were sent 
quarterly with follow-up telephone interviews regarding message receipt and topic recall conducted 5-10 days 
after the message delivery date. During a pandemic when numerous messages are sent, alert fatigue may impact 
ability to recall whether a specific message has been received due to the "noise" created by the higher number of 
messages. To determine the impact of "noise" when study messages were sent, we compared health care provider 
recall of the study message topic to the number of local public health messages sent to health care providers. 

Results: We calculated the mean number of messages that each provider received from local public health during 
the time period around each study message and provider recall of study message content. We found that recall 
rates were inversely proportional to the mean number of messages received per week: Every increase of one local 
public health message per week resulted in a statistically significant 41.2% decrease (p < 0.01), 95% CI [0.39, .87] in 
the odds of recalling the content of the study message. 

Conclusions: To our knowledge, this is the first study to document the effects of alert fatigue on health care 
providers' recall of information. Our results suggest that information delivered too frequently and/or repetitively 
through numerous communication channels may have a negative effect on the ability of health care providers to 
effectively recall emergency information. Keeping health care providers and other first-line responders informed 
during an emergency is critical. Better coordination between organizations disseminating alerts, advisories and 
other messages may improve the ability of health care providers to recall public health emergency messages, 
potentially impacting effective response to public health emergency messages. 
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Background 

"Information chaos" — various combinations of informa- 
tion overload, information underload, information scat- 
ter, information conflict and erroneous information — is 
an increasingly cited factor affecting the effectiveness, 
quality and safety of clinical care [1]. By contributing to 
increased mental workload and decreased situational 
awareness in the work place, information chaos may also 
be a factor in alert fatigue; i.e., health care providers 
(HCPs) missing or ignoring important messages within 
the volume of information they have been conditioned 
to perceive as irrelevant [2]. Previous alert fatigue stu- 
dies have focused on clinical decision support systems 
(CDS) and computerized provider order entry (CPOE) 
systems which aim to improve efficiency and quality of 
care. These studies have investigated how CDS and 
CPOE systems alert providers to potential adverse medi- 
cation events and interactions [3,4], the frequency and 
factors associated with overriding CPOE alerts [2,5], 
recommended design approaches for mitigating adverse 
drug events and alert fatigue [6,7], and unintended conse- 
quences of these systems in contributing to information 
and mis-information overload, role confusion, excessive 
errors and alert fatigue [8]. In addition to the clinical 
system messages, HCPs receive public health alerts and 
advisories, clinical guidelines and updates, and training 
notifications from professional associations, agencies 
and organizations [9] and through a system of national, 
state and local public health communication channels [10]. 

HCPs serve as partners with public health in surveil- 
lance and case investigation activities by providing case 
reporting and clinical management, preventing excess 
deaths, treating the injured and mitigating suffering 
[11]. For example, HCPs were the first to recognize 
early cases of inhalation anthrax in 2001 [12] and first 
to identify dengue outbreaks in Hawaii, Florida and 
Texas between 2001-2011 [13]. In an emergency, pub- 
lic health agencies rely on HCPs as both frontline re- 
sponders and as trusted and preferred communicators 
of health information to the public during an emer- 
gency [14-16]. 

Ensuring that timely, accurate, trustworthy and con- 
text-relevant information can be "heard" by HCPs within 
their information chaotic environments is essential to an 
effective public health communication program [17,18]. 
However, while their intention is to reduce information 
chaos, the growing variety of public health agency and 
other messaging systems may be increasing, rather than 
reducing, communication challenges for HCPs [9]. Poor 
coordination between agencies can contribute to in- 
consistent and/or conflicting information as well as re- 
dundant messaging between public health agencies and 
HCPs, as demonstrated by communication issues identi- 
fied in the 2008 Salmonella Saint Paul outbreak response 



[19]. Similar issues were recently documented by a 
process analysis and survey conducted with primary care 
providers in Utah following the first wave of the 2009 
influenza pandemic. A majority of providers stated that: 
1) they received an overwhelming amount of email from 
public health authorities as well as health care organiza- 
tions and 2) they desired less frequent, more concise 
and locally relevant emergency communications during 
a public health emergency [20]. Despite recognition of 
these problems, little is known about the potential for 
HCP alert fatigue from public health communications. 

The reach (Rapid Emergency Alert Communication in 
Health) study is a randomized controlled trial to system- 
atically evaluate and compare the effectiveness of mobile 
(SMS) and traditional (email, fax) communication strat- 
egies for sending public health messages to health care 
providers — physicians, pharmacists, nurse practitioners, 
physician assistants and veterinarians. The study aims to 
identify which communication modality is most effective 
for dissemination of health alerts and advisories between 
public health agencies and HCPs in order to improve 
emergency preparedness and response. The findings 
reported in this paper are a reach sub-study involving 
one study site. The objective of the analysis described 
in this paper is to measure the effects of public health 
message volume on recall of specific message content 
as a measure of HCP alert fatigue. 

Methods 

Ethics 

Study protocols were approved by the UW Institutional 
Review Board (Minimal Risk Category 7). 

Design and setting 

The reach study is a multi-site randomized control trial 
conducted by the University of Washington (UW) across 
three sites in partnership with local (King County, WA 
and Spokane County, WA) or state (State of Montana) 
public health agencies. The findings reported in this 
paper are a reach sub-study involving the King County, 
WA study site. Between October 2009 and January 
2011, the reach study was conducted in King County, 
WA, the most populous county in Washington State, 
and the 14th most populous in the United States [21]. 
HCPs were recruited and randomized to receive public 
health messages via email, fax, short message service 
(SMS delivered to a cell phone) or to a control group 
that did not receive messages. HCPs were blindly ran- 
domized regardless of ability to receive the group inter- 
vention; for example, HCPs who did not provide a fax 
number at enrollment could be blindly randomized into 
the fax group. Investigators, data analysts and inter- 
viewers were blinded to randomization groups. 
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Eligibility 

Providers were eligible for the study if they were an ac- 
tive, practicing physician in a primary care related spe- 
cialty, nurse practitioner, pharmacist, physician assistant 
or veterinarian in King County, WA. Providers were ex- 
cluded if they were inactive or retired, involved with the 
study or worked for the site's public health agency. Phy- 
sicians were oversampled due to historically low study 
response rates from this population. Of 7,281 eligible 
HCPs in King County, 530 enrolled in the study and 528 
were included in this analysis. Table 1 provides HCP 
demographics. 

Intervention 

Between March 2010 and January 2011, messages based 
on real events of public health significance were sent 
quarterly to enrolled HCPs. All messages included a link 
to a public health web page with additional information. 
Message format varied by delivery mechanism as seen 
in Figure 1 Email and fax messages included standard 
content while those delivered via SMS were limited to 
160 characters. 

Intervention assessment 

Follow-up phone interviews were conducted 5-10 days 
after message delivery date. Interviews consisted of ap- 
proximately six questions that elicited information 
about message receipt, recall of its content and per- 
ceived credibility and trustworthiness of the message 
and its source. Figure 2 illustrates the interview proto- 
col. The interview included two methods for determin- 
ing whether HCPs who reported receiving the message 
recalled its topic correctly (boxes outlined in red on 
Figure 2): unprompted recall and prompted recall. 
Unprompted recall measured whether the HCP correctly 
identified the reach message either alone or in combin- 
ation with another message topic. If HCPs recalled 
receiving a message but identified a non-reach topic, 



Table 1 Demographic characteristics of study participants 
in King County 





N 


% 


Mean 


SD 


Male/ 
Total 


% Male 


Provider 
type 






Age 




Gender 




ARNP 


126 


23.86 


48.6 


10.5 


12/121 


9.9 


MD 


209 


39.58 


48.8 


9.5 


106/200 


53.0 


PA 


35 


6.63 


46.0 


11.0 


10/33 


30.3 


PHRM 


103 


19.51 


45.9 


12.4 


39/93 


41.9 


VET 


55 


10.42 


44.7 


11.0 


16/53 


30.2 


Missing 


0 




64 




28 





Provider Type: ARNP Advanced Registered Nurse Practitioner, MD Physician, 
PA Physician Assistant, PHRM Pharmacist, VET Veterinarian. 



interviewers utilized the prompted recall script. In that 
script, HCPs were asked first if they could recall receiving 
a message about a topic which was not related to the con- 
tent of any messages sent by the reach study or the public 
health agency (i.e., a "fake" or distractor topic) and then if 
they remembered the study message topic. HCPs who did 
not remember receiving a public health message in the 
previous two weeks, gave an open-ended response that did 
not match the study message topic or claimed to remem- 
ber the "fake" topic were classified as not correctly 
recalling the message. HCPs that correctly remembered 
the reach study message topic with and without prompting 
were classified as having correcdy recalled the message. 

"Noise" data collected 

To calculate "noise" during the timeframe of each reach 
message, HCPs who subscribed to the public health 
department's email listservs were identified and the 
number and topic of messages disseminated by the pub- 
lic health department's listserv 4 weeks prior to and 3 
weeks after each study message were documented. Of 
the sample total of 528, 23.7% of HCPs were identified 
as subscribers to at least one public health listserv (see 
Table 2). Public health department messages included a 
range of alerts, advisories, guidelines and subscription 
information. Figure 3 illustrates the variation in number 
of public health messages sent per month prior to and 
during the study period. 

Data analysis approach 

All analyses were conducted using R v2.10.0 [22]. Basic 
HCP demographic data were compiled based on HCP 
information provided at enrollment and interview ques- 
tions regarding age and gender (Table 1). Proportions of 
HCPs recalling receipt of study messages and correctly 
recalling study message topics with and without promp- 
ting were calculated using responses from follow-up 
interviews. 

Data analysis methods 

To examine the main effect of message rate on alert 
fatigue, we analyzed the effects of message rate on the 
probability of correctly recalling content of study mes- 
sages among those who remembered receiving messages. 
A HCP was classified as having correctly recalled the 
reach study message topic if the HCP: 1) Recalled receiv- 
ing a message without prompting and the topic correctly 
matched the reach study message topic; or 2) Did not 
recall receiving the "fake" topic but did recall the reach 
study message topic. To discount the effects of "agree- 
ability bias", HCPs who recalled receiving a message on 
the "fake" topic and required prompting for the study 
message topic were not counted as correctly recalling 
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Figure 1 Example reach study message: Full message (left) sent via email and fax and truncated messages (right) sent via SMS. 

All messages contained a link to a web page (center). 



Did you receive a public hearth alert 
in the last 2 weeks? 



Did you receive any 
public health alert in the 
last 2 weeks? 



How did you receive the alert? 



Unprompted Recall: 
Do you recall the content or subject of the alert? 



YES 
Mentions 
ALERT TOPIC 
only. 



YES 

Mentions ALERT 
TOPIC AND 
another topic. 



YES 

Mentions NON- 
ALERT TOPIC. 



No/Do not 
know 



Do you remember receiving an alert about 
(FALSE TOPIC1 in the last two weeks? 



YES/No/Do not know/Refuse 



No/Do not 
know/Refuse 



End of Survey 



Prompted Recall: 
Do you remember receiving an alert about 
TOPIC in the last two weeks? 



ED 



Figure 2 Phone interview questions used to measure outcome measures of message recall 



No/Do not 
know/Refuse 
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Table 2 Listserv subscription of King County health 
care providers 



Listserv subscription? 


N 


% 


Yes 


125 


23.67 


No 


403 


76.33 


Missing 


0 






Mean 


SD 


Messages received per week from listserv* 


0.21 


0.54 


Missing 


0 





* For periods prior to communication of study messages. 

receipt of the study message. All other responses were 
classified as not correctly recalling the message. 

Analysis was conducted using generalized linear mo- 
dels (GLM) with Binomial outcomes and the logit link 
function with exchangeable autocorrelations within pro- 
viders using generalized estimating equations (GEE) in 
using the function 'gee' in the 'gee' package. Robust 
standard errors were used to calculate z scores and p- 
values. Data from the first message were not included in 
this analysis because questions prompting HCP recall 
with the study message topic were only used on inter- 
views for messages 2-5. Message number was included 
in the model to control for differences in recall or 
"memorability" of the study message topic. 

Exploratory data analysis of the data indicated that 
one of the (blinded) randomization groups was obviously 
the control group due to a large failure rate in recall. 
For the alert fatigue analysis and results that follow, the 
control group was excluded since the alert fatigue 



analysis focused on an HCP's ability to recall whether 
our message was received amidst the "noise" of other 
messages they received and there was no mechanism 
by which message rate could influence recall of the 
study messages. Two covariates other than message 
rate — message number and (blinded) randomization 
group — were included in the main analysis. Our ana- 
lysis investigated: 1) HCP rates of message recall for 
both prompted and unprompted responses; 2) esti- 
mated effect of increases in message rate on prompted 
recall; and 3) differences in the effect of message rate 
on recall between provider types, gender, age and 
randomization group (message delivery method). 

Results 

Message recall 

We found substantial variation in the rates at which HCPs 
recalled the study message topic for both prompted and 
unprompted responses (see Table 3 for rates of recall 
for each message). Recall rates were inversely proportional 
to the mean number of messages received per week. A 
substantial proportion (28.1%) of HCPs who recalled 
the study message topic after prompting also recalled 
the "fake" topic; this did not vary substantially across 
messages. 

Effect of message rate on alert fatigue 

Analysis of prompted recall (Table 4) was calculated 
using a generalized estimating equation estimation of a 
generalized linear model (binomial family with logistic 
link function) with prompted recall as the outcome and 
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Figure 3 Number of messages sent over public health agency email listserv per month from 2008-2010. The two waves of the 2009 
H1N1 pandemic are marked by dramatic increases in messages. 
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Table 3 Rates of message recall by message and outcome type 



Outcome 


Message 
2 


Message 
3 


Message 
4 


Message 
5 


Mean number of messages received per week by participants belonging to any listserv. 


1.13 


0.33 


0.35 


0.68 


Recall content of study message (coded open ended question) among those that 


66/171 


139/181 


106/191 


118/212 


could recall content 


66/171 


139/181 


106/191 


118/212 


"FakeVDistractor topic recalled when study topic was recalled 


32.2 


27.4 


26.1 


29.3 




19/59 


20/73 


24/92 


17/58 


Correct recall of study message by prompting or by open ended response among 


52.7 


74.9 


62.5 


57.3 


participants who recall receiving any message 


87/165 


1 76/235 


1 50/240 


1 34/234 


Correct recall of study message by prompting or by open ended response, over 


24.4 


42.4 


36.6 


35.6 


all participants interviewed 


87/356 


176/415 


1 50/41 0 


134/376 



weekly message rate and message number as covariates. 
We found that every increase of one message per week 
resulted in a statistically significant 41.2% decrease in 
the odds of recalling the content of the study message 
(p < 0.01), 95% CI [0.39, .87]. Within HCP correlation 
was small at 0.10. Differences in the effect of message 
rate on recall between provider types, provider gender 
and provider age were examined using analysis of 
covariance of generalized linear models by separately 
including each covariate as an interaction term with 
message rate. We detected no effect of message rate on 
recall between provider types, gender or age. 

The inclusion of the additional interaction term re- 
duced the residual deviance by 0.38, 2.80 and 0.08 
respectively on 4, 1 and 1 degrees of freedom; these re- 
ductions are all small in magnitude and none are statisti- 
cally significant. An analysis of randomization group on 
alert fatigue demonstrated no effect of communication 
channel on alert fatigue. 

Discussion 

To our knowledge, this is the first study to document 
the effects of alert fatigue on HCPs' recall of information. 
The number of local public health messages significantly 
decreased the odds of HCPs correctly remembering the 
receipt and content of a study message, and suggests 



Table 4 Analysis of prompted recall 



Covariates 


Effect 
(odds scale) 


Coefficient 


Robust 
standard error 


Robust 
z-score 


(Intercept) 




0.755 


0.227 


3.332 


weekly message 
rate 


0.588 


-0.531 


0.209 


-2.545 


Message 3 


1.853 


0.617 


0.261 


2.361 


Message 4 


0.999 


-0.001 


0.265 


-0.005 


Message 5 


1.037 


0.036 


0.244 


0.147 



Within participant 
correlation: 0.10 



that alert fatigue and information overload may inhibit 
the ability of HCPs to respond effectively to messages 
during a public health emergency when volume of mes- 
sages may be higher. The higher than expected recall of 
a "fake" topic, despite its irrelevance to any recent pub- 
lic health message, may indicate a degree of HCP role 
confusion or uncertainty around public health messages 
or it could be an outcome related to information chaos 
in the health care setting. 

Information delivered too frequently and/or repetitively 
through multiple communication channels also may ne- 
gatively affect the ability of HCPs to recall public health 
messages. It should be noted that six months prior to the 
date of this study, King County HCPs on local public 
health listservs were exposed to high volumes of health 
alerts and advisories during the initial onset of the 2009 
H1N1 pandemic. This "noise" may have desensitized HCPs 
to information delivered through the public health emer- 
gency message system. We also did find a general trend 
of improvement in recall of message receipt and recall 
of content over time, which could be attributed to re- 
duced message volume in the time period before the 
study message and could indicate that HCPs recovered 
from the high volume of messages sent during the 
H1N1 pandemic. 

It might seem logical that an electronic health record 
(EHR) that provides contextually relevant information to 
clinicians would also provide a delivery mechanism for 
population-based information. However, it is far from 
clear that this information would not become part of the 
clinical "noise" and contribute to increased information 
chaos in the clinical environment. Alert fatigue studies 
conducted in clinical environments have documented 
that over 90% of medication safety/drug-drug interaction 
alerts are overridden in electronic decision support sys- 
tems without a physician even looking at the alert [23]; 
pharmacists have also been found to override one-third 
of life-threatening drug-drug interactions [24]. Low alert 
specificity, unclear information content, perceived lack 
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of relevance, information overload and unnecessary dis- 
ruptions to workflow are cited as reasons for ignoring 
alerts [2]. While the opportunity to leverage EHRs to 
disseminate public health alerts and advisories is promis- 
ing, more research is needed to identify the most effect- 
ive means of delivering these messages to HCPs, as well 
as how to communicate with HCPs who do not work in 
an environment with an EHR. 

Also more research is needed to identify public health 
message sources of greatest value to HCPs. Anecdotal 
evidence from open-ended interview responses sug- 
gested that at least some HCPs received messages from 
other sources, such as the CDC, FDA, FEMA, as well 
as professional organizations such as the Washington 
State Medical Association and internal employer commu- 
nications during the study period. We were unable to 
accurately document these additional message sources, 
although source of message may impact the likelihood 
of message recall. 

Limitations 

Our study has several limitations. 

Study messages, while timely, were not emergency 
alerts and, thus, may have been perceived as less rele- 
vant, important or "memorable" than other public health 
agency messages HCPs received during the same time 
frame. In addition, we varied the day of the week that 
study messages were sent. It is possible that messages 
sent on certain days are better recalled by HCPs and that 
our messages were sent on lower recall days. To study 
this phenomenon would require communication of mul- 
tiple study messages on each day of the week — a wholly 
different study than we conducted — thus we do not have 
data to determine whether HCPs experience lower or 
higher recall days. 

The primary explanatory variable of interest, message 
rate, is an uncontrolled variable. Our ability to estimate 
the effect of message rate depends on our ability to 
separate out the effects of alert fatigue from the "mem- 
orability" effects of individual messages. This depends 
on the variability in the number of messages received 
between participants, and a large part of this variability 
comes from differences based on public health listserv 
membership. To the extent that systematic differences 
exist between providers who do and do not belong to 
public health listservs, biases may exist in our estimates. 
We identified HCPs who subscribed to public health 
listservs by comparing listserv and study contact infor- 
mation. It is possible that our listserv identification was 
not accurate as HCP contact information may have 
changed or HCPs may have subscribed to a listserv after 
the study began, which in turn means our estimates of 
the rate at which messages were received by participants 
may have been underestimated. The effect of such bias 



would be to underestimate the effect size of message 
rate on message recall. 

We also are unable to document whether reach subjects 
actually received the local public health listserv communi- 
cations or whether a listserv message was shared with 
other HCPs; we can only document that a public health 
listserv message was sent to the HCP. This highlights an 
issue for public health messaging through lists — whether 
email, fax or SMS — in general. Most public health listservs 
require HCPs to opt in to receiving messages and the 
burden of maintaining correct contact information rests 
on public health agency staff. A clinic, hospital or other 
health care organization that maintains a directory of pro- 
vider contact information may be more a more efficient 
mechanism for disseminating public health messages 
and communicating emergency information to its HCPs. 
However, this raises the question of whether a public 
health message has the authority or novelty to be "heard" 
within the usual clinical noise and would need further 
investigation. 

While this study contributes to the evidence base for 
improving public health agency emergency communica- 
tions with HCPs, we cannot correlate failure to recall 
message topic with inability to act on the information 
received, such as HCPs communicating information fur- 
ther downstream by, for example, sharing email with 
colleagues or disseminating information to patients or 
the public. Recent disasters have highlighted effective 
communication as essential in public health agencies' 
ability to protect populations during emergencies; these 
events can increase information chaos in the health care 
environment. More research is needed to understand 
the impacts of the method of message delivery, the 
source of the message, the type of information being 
disseminated and the health care organizational context 
to guide or improve the practice of communication be- 
tween public health agencies and HCPs before, during 
and after a public health emergency. 

Conclusions 

Keeping health care providers and other first-line re- 
sponders informed during an emergency is critical. Im- 
plementing measures to avoid redundant and conflicting 
messages from multiple agencies may help reduce alert 
fatigue. Delivering public health alerts and advisories too 
frequently and/or repetitively may have a negative effect 
on the ability of HCPs to effectively recall and, poten- 
tially, to respond to public health emergency messages. 
Our results suggest that public health agencies should 
consider the potential impacts of timeliness of informa- 
tion and frequency of communications in emergency 
planning to reduce the potential for alert fatigue and 
information chaos during public health emergencies. 
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